Monitoring key-press delay and duration to determine need for assistance

ABSTRACT

In a user-interactive system, such as an interactive voice response (IVR) system, the duration and/or force of key-presses by the user and/or the speed of response to prompts by the user are used to make inferences about the user, such as whether the user is impaired or is growing frustrated. The user interface is modified based on the inferences.

TECHNICAL FIELD

This invention relates generally to interactive systems, andspecifically to interactive communications systems.

BACKGROUND OF THE INVENTION

Callers who experience difficulty with an interactive voice response(IVR) system may become so frustrated that they hang up beforeaccomplishing their objective. This can result in toward, and lostrevenue for, the called party. Thus, there is a need to detect whencallers are having trouble, and intervene on their behalf. The likeconsiderations exist not only in IVR systems, but in any systems thatrequire interaction with a user, such as Automatic Call Distribution(ACD) systems, for example.

In this context, users with cognitive and/or physical disabilities mighthave a harder time navigating a typical IVR or other interactive userinterface than the general population. Therefore, disability accessrules enforced by the FCC require support for people who are cognitivelyimpaired. Specifically, the Code of Federal Regulations, 36 C.F.R. Part1193.41(i), states the following: “Input, control, and mechanicalfunctions shall be operable with limited cognitive skills. Provide atleast one mode that minimizes the cognitive, memory, language, andlearning skills required of the user.” This regulation implies that IVRsystems should adjust their behavior to accommodate the needs ofcognitively impaired users. Unfortunately, it is often not practical—andperhaps even illegal—to maintain a database that identifies specificcaller IDs or specific accounts as belonging to someone who has adisability. The problem then is, how can an IVR system detectautomatically that a user is impaired? A way of detecting need forassistance is to monitor a user's interaction with the system. Indeed,some IVR systems monitor key-press sequences in order to detect that auser is having problems. For example, it may be assumed that users needadditional help when they frequently ask for menus to be repeated, or ifthey are observed to go down a branch of a menu, come back up, go down adifferent branch, and then come back up again. Key-press duration hasbeen used to determine the action to be taken. For example, on anelectric typewriter, it is common for the “key repeat” function to betriggered by holding down a key longer than is necessary for a singlecharacter to be typed. And on electronic watches, it is common for aprogramming mode change to be triggered by holding down a button. Suchtime-dependent controls are explicit components of the user interface.In the mid 1980s, studies were conducted by Telcordia Technologies (thennamed Bell Communications Research, Inc.) to determine whethermeasurements of inter-digit pauses while personal identification numbers(PINs) are being entered might help to detect when the PINs are beingentered by someone other than their owners.

SUMMARY OF THE INVENTION

According to one aspect of the invention, the duration of actuations ofan actuator (e.g., key-presses) by a user are used to make inferencesabout the user, such as whether or not the user is having difficultywith the user interface and needs assistance. According to this aspect,the duration of actuation of an actuator is determined; in response to along duration, the user interface is modified; and in response to anot-long duration, the interface is not so modified. According toanother aspect of the invention, the speed of response to prompts (i.e.,the delay between prompts to a user and the user's actuations of anactuator in response to those prompts) is used to make inferences aboutthe user. According to this aspect, the speed of response to a prompt isdetermined; in response to a slow speed, the user interface is modified;and in response to a not-slow speed, the interface is not so modified.According to yet another aspect of the invention, the force, pressure,of actuation of an actuator is also used to make inferences about theuser, such as whether the user is growing frustrated. According to thisaspect, the force of actuation of an actuator is determined; in responseto a high force, the user interface is modified; and in response to anot high force, the interface is not so modified. Preferably, a systemuses the duration of the actuation, the force of the actuation, and/ordelay data, in combination with other metrics (for example, the numberof times that a user asks for a menu to be repeated) in order to improvethe accuracy of the inference.

The user interface is adjusted based on the inferences. For example, ifit is detected that a user's key-press durations change during a singleIVR session from a normal level to an extended level, it is inferredthat the user is having trouble or is growing frustrated, and the systemcan respond by transferring the call to a human agent. Or, if it isdetected that the key-press durations are extremely long at the start ofthe session, and remain long during the session, it is inferred that theuser has a disability of some sort. This could be a physical disability,such as a motor-skill impairment, or a cognitive disability. It ispossible to distinguish between the two by examining other metrics; forexample, frequent requests for menus to be repeated is assumed toindicate cognitive or language issues, rather than a physicaldisability. Or, the user can be questioned in order to determine thenature of the user's disability. On the one hand, when a cognitive orlanguage problem is assumed to be present, the IVR system can adjust theuser interface by, for example, slowing down the playback rate of thespeech, adjusting the menu structure so that fewer options are presentedat each node (thereby making the available options easier to remember),explaining each option in greater detail, and/or requesting confirmationof the user's entries from the user. On the other hand, an accommodationfor people with motor impairments is an automatic lengthening of thecall-terminating time-out period, or of the system-imposed inter-digittime-out period, when users are entering a series of characters (e.g.,dialing a phone number).

Conversely, if it is detected that a user's key-presses have unusuallyshort durations, it may be inferred that the user is an experienced userwho is familiar with the interface, and the system can respond by makingthe interface navigable by the user more quickly, such as by giving theuser terse prompts and more choices at each prompt, and playing theprompts at a faster speed.

Although these examples describe telephony applications, the idea isuseful in other domains, such as with interfaces to portable electronicdevices and to computers, and with Internet applications. For example,the technique is as useful when applied to button-press durations on amouse or stylus-press durations on a computer as it is for dial-padkey-press durations.

While the invention has been characterized above as a method, it alsoincludes apparatus that performs the method, as well as acomputer-readable medium containing instructions which, when executed bya computer, cause the computer to perform the method.

BRIEF DESCRIPTION OF THE DRAWING

These and other features and advantages of the invention will becomemore apparent from considering the following description of anillustrative embodiment of the invention together with the drawing, inwhich:

FIG. 1 is a block diagram of a system that includes an illustrativeembodiment of the invention;

FIG. 2 is a functional flow diagram of a first illustrative embodimentof an interface application of the system of FIG. 1; and

FIG. 3 is a functional flow diagram of a second illustrative embodimentof the interface application of the system of FIG. 1.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of a system that includes an illustrativeembodiment of the invention. The system includes a key entry device 116by means of which a user enters information by pressing keys, a timingdevice 110 for timing key-press durations and/or delays, and aresponsive device 100 that responds to the key-press durations and/ordelays. Device 100 is typically computerized and comprises a processor120 and a memory 122 or any other computer-readable medium storing dataand/or programs for use by processor 120, 1 including an interfaceapplication program 122 that implements a user interface 125 for a userof key entry device 116.

System 100 is described generically, and may take many different formsin actual implementation. For example, the system of FIG. 1 may be atelecommunication system where key entry device 116 is the dial keypadof a telephone, timing device 110 is a timing function of a telephoneswitch, and responsive device 100 is an IVR system. Or, the system ofFIG. 1 may be a computer or a personal digital assistant (PDA) where keyentry device 116 is a keyboard, mouse, stylus, and/or a touch-sensitivescreen, timing device 110 is a timing function of the computer or PDA,and responsive device 100 comprises a display of the computer or PDA.Or, the system of FIG. 1 may be a game controller where key entry device116 is a joystick and button game control, timing device 110 is a timingfunction of the game controller, and responsive device 100 is thespeaker and display or display output of the game controller. These arejust a few illustrative examples of the myriad forms that the system ofFIG. 1 can take.

FIG. 2 shows functionality of one illustrative embodiment the system ofFIG. 1 where key entry device 116 is a wireless phone and responsivedevice 100 is an IVR system executing interface application 124. Thewireless phone detects the speed of its user's response to IVR prompts,the inter-digit intervals between key-presses, and the key-pressdurations and force, at step 200, and reports them to the IVR system,for example, via the tone-duration field of system messages of the IETFRPC 2833 signaling protocol. The IVR system compares the response speedand inter-digit intervals against historical records for either thisuser or the population at large to determine if the response speed isslow and inter-digit time is long, at step 202. If so, the IVR systemcompares the key-press durations and force against historical records todetermine if the durations are long or the force is high, at step 204.If so, the system infers that the user is distracted, perhaps because heor she is using the telephone while driving, and therefore it lengthensthe inter-digit time-out and call-terminating time-out periods, at step206. Returning to step 204, if the IVR system determines that thekey-press durations are long or forceful, it infers that the user isimpaired, and therefore it simplifies the menus that it presents to theuser, at step 210, and offers assistance of an automatic speechrecognition (ASR) interface to the user, at step 212.

Returning to step 202, if the IVR system determines that the responsetime and the inter-digit time are not slow, it compares the key-pressdurations and force against historical records to determine if thedurations are long or the force is high, at step 214. If so, the systeminfers that the user is frustrated, and therefore connects the user to ahuman agent, at step 216. If the IVR system determines at step 214 thatthe key-presses are not long or forceful, it infers that the user needsno assistance with the user interface, and therefore it continues topresent the standard, unmodified, interface to the user, at step 218.

FIG. 3 shows functionality of a second illustrative embodiment thesystem of FIG. 1 where key entry device 116 is a fixed phone andresponsive device 100 is an IVR system executing interface application124. The switching system to which the phone is connected acts as timingdevice 110 and monitors the user's key-press durations, at step 300,which it then reports to the IVR system. The IVR system stores theduration information. If the IVR system detects that the key-pressduration is consistently longer-than-normal, at step 302, it comparesthe newly-arriving duration information against the stored informationto determine if the durations of key-presses are progressively gettinglonger, at step 304. If so, the IVR system infers that the user isgetting frustrated, and so it connects the user to an agent, at step306; if not, IVR system queries the user to determine what the problemmay be, at step 310. For example, it may query whether it would help theuser to slow down the playing speed of the menu prompts. It might alsoquery the user whether fewer options per menu would be helpful. Othervariations in the user interface could be offered and implemented basedon feedback from the user. It then uses the user's responses todetermine if the user is physically or cognitively impaired, at step312. If the user is believed to be cognitively impaired, the IVR systemchanges the prompt-menu structure to present simpler choices to theuser, and also slows down the playback speed of the prompts, at step314. The actual difference between a distracted user and one withcognitive impairment may not matter in terms of the options offered tothe user, since both can require the same action. The IVR system alsostarts to provide confirmations of the user's responses to the user, atstep 316. If the user is found, at step 312, to have a physicalimpairment, the IVR system queries the user for whether he or she wouldprefer to use an ASR interface, or to speak to an agent, at step 318.

Returning to step 302, if the key-press durations are not consistentlyfound to be longer-than-normal, the IVR system checks whether thekey-press durations are consistently shorter-than-normal, at step 330.If so, the IVR system infers that it is interacting with an experienceduser, or one that has normal abilities and is in a hurry, and so itcompresses the prompts that it gives to the user, at step 334, forexample, by making the prompts terser, presenting more choices in eachannouncement, and increasing the playback speed of each announcement. Ifthe IVR system keeps historical records on past interactions with users,the IVR system may also offer to skip the menu and short-cut the user toa cached menu of the user's favorite choices, at step 336. If the IVRsystem finds at step 330 that the key-press durations are notconsistently shorter-than-normal, it just continues to present thestandard interface to the user, at step 332.

Of course, various changes and modifications to the illustrativeembodiment described above will be apparent to those skilled in the art.For example, other environments in which the invention may beadvantageously employed include interacting with an ACD system, a gamewhere the controller adjusts the sensitivity of the joy stick and/or theuser feedback based on the input observed at a previous level in thegame so that the game learns to be tougher as the user's skills at thegame gets better, interacting with a burglar alarm keypad where the usergets better over timer at the use of the system so as to benefit fromspeed-up but where the user on occasion may be slower than typicalbecause he or she is carrying something, and interacting with a TVremote control where detection that the operator is having problems mayresult in the TV screen being used to display to the operator the stepsinvolved in using the remote to reach a particular program and/oractivation of an easier interface that requires the operator to use justthe arrow keys and an “enter” button. Such changes and modifications canbe made without departing from the spirit and the scope of the inventionand without diminishing its attendant advantages. It is thereforeintended that such changes and modifications be covered by the followingclaims except insofar as limited by the prior art.

1. A method comprising: determining a duration of actuating an actuatorby a user; determining if the duration is longer than a first time; ifthe duration is longer than the first time, determining that the user isimpaired; in response to determining the user is impaired, modifying auser interface interfacing with the actuator to compensate for theimpairment; and in response to determining that the user is notimpaired, forbearing from modifying the user interface.
 2. The method ofclaim 1 wherein determining if the duration is long comprises:determining, over time, durations of a plurality of actuations ofactuators by a user; comparing the determined durations to determine ifthe durations are getting longer over time; and in response todetermining that the durations are getting longer over time, modifyingthe user interface of the user.
 3. The method of claim 1 whereinmodifying the user interface of the user comprises: connecting the userto an agent.
 4. The method of claim 1 wherein determining that the useris impaired comprises: interacting with the user to determine if theuser is impaired.
 5. The method of claim 1 wherein determining that auser is impaired comprises: interacting with the user to determine ifthe user has a physical or a cognitive impairment in response todetermining that the user has a physical impairment, modifying the userinterface in a first manner; and in response to determining that theuser has a cognitive impairment, modifying the user interface in asecond manner.
 6. The method of claim 1 further comprising: determiningwhether the user is frustrated; in response to determining that the useris frustrated, modifying the user interface in a first manner;determining whether the user is impaired; and in response to determiningthat the user is impaired, modifying the user interface in a secondmanner.
 7. The method of claim 1 wherein modifying comprises: modifyingthe user interface in a first manner; forbearing comprises: in responseto a short said duration, modifying the user interface in a secondmanner; and in response to a normal said duration, forbearing frommodifying the user interface.
 8. The method of claim 1 whereindetermining if the duration is longer than a first time comprises,determining a speed of response to a prompt by the user; modifyingcomprises: in response to a long said duration and a slow said speed ofresponse, modifying the user interface; and forbearing comprises: inresponse to a short said duration and a fast said speed of response,forbearing from said modifying.
 9. The method of claim 1 whereindetermining if the duration is longer than a first time comprises:determining intervals of time between successive said actuations; anddetermining an interval of time between a prompt and a first saidactuation; modifying comprises: in response to long said durations andlong said intervals, modifying the user interface; and forbearingcomprises: in response to short said durations and short said intervals,forbearing from said modifying.
 10. The method of claim 1 wherein thefirst time is a historic average of the duration of actuating theactuator.
 11. An apparatus comprising: a user interface operable tointerface with a user; a device in communication with the userinterface, the device comprising a processor and memory, the deviceoperable to: modify the user interface in response to a change in aduration of actuating of an actuator connected to the user interface;and forbear from said modifying in response to no change in the durationof the actuation.
 12. The apparatus of claim 11 wherein: the devicecomprises an interactive voice response system; and the actuator isincluded in a keyboard of a user terminal device.
 13. The apparatus ofclaim 11 wherein: the device comprises a wireless phone; and theactuator is included in a dial pad of the phone.
 14. The apparatus ofclaim 11 wherein: the device comprises a computer; and the actuator isincluded in a keyboard or a mouse of the computer.
 15. The apparatus ofclaim 11 wherein: the device comprises a personal digital assistant; andthe actuator is included in a keyboard or a touch-sensitive screen ofthe personal digital assistant.
 16. The apparatus of claim 11 wherein:the device comprises a burglar alarm system; and the actuator isincluded in a keypad of the burglar alarm system.
 17. The apparatus ofclaim 11 wherein: the device comprises an electronic game; and theactuator is a hand control of the game.
 18. The apparatus of claim 11wherein: the device comprises a television; and the actuator is includedin a remote control of the television.
 19. The apparatus of claim 11wherein the device further comprises: a key entry device having at leastone pressable key; a timing device for determining durations of pressingof the at least one key; and a responsive device for selectivelymodifying the user interface in response to the determined durations.